Section: New Results

Optimization Approach for Deep Neural Network Training

Participants: Emilie Chouzenoux, Jean-Christophe Pesquet, Vyacheslav Dudar (in collaboration with G. Chierchia, Univ. Paris Est and V. Semenov, Univ. of Kiev)

In paper [28], we develop a novel second-order method for training feed-forward neural nets. At each iteration, we construct a quadratic approximation to the cost function in a low-dimensional subspace. We minimize this approximation inside a trust region through a two-stage procedure: first inside the embedded positive curvature subspace, followed by a gradient descent step. This approach leads to a fast objective function decay, prevents convergence to saddle points, and alleviates the need for manually tuning parameters. We show the good performance of the proposed algorithm on benchmark datasets.